Indraneel's blog

Things that every person looking to run a service must consider

July 4, 2014

Regardless of the number of consumers of your service, there are a few commonalities when considering how to build a service. This is my attempt at listing the ones that stand out for me. I will follow up this list with detailed write ups expanding on each of these topics later

Have a deployment script

Whether you run a single person service or a massive operation, being able to deploy repeatedly with out needing manual steps should be the top priority. The same script should preferably be able to deploy to any target starting from a developer's box all the way upto the datacenter or your choice of public cloud.

Version and live patch the service

You will need to patch your service and sooner or later you will have to patch it with out downtime. Most live patching strategies discuss 2 concepts

• Splitting up the service constituents and patching them in independently in sequence: This allows for the constituents that have already been patched to gloss over the glitches in the service constituents that are currently being patched. Glossing over becomes much easier when the service onstituents are properly versioned.
• Chunking a given constituent into subsets and patching them sequentially: This allows for the constituent being patched to maintain the ability to respond to incoming requests because something is always available.

Have a configuration system

Any reasonably large service has numerous knobs that need tweaking during deployments. Get the developers to agree to a central configuration file/store which is where the variables are centralized allowing for sharing and reuse. Good configuration systems provide an easy way to add variables and query their values in numerous ways.

Have a story for managing secrets and certificates

Eventually, any serious service needs to provide SSL support, possibly data encryption and very often interact with other services. You end up needing a few number of certificates and often a few number of passwords. It is tempting to think of certificates and secrets either as configuration (and therefore become insecure), or as part of infrastructure (secure but a pain to manage), but any good service ends up having to ensure that they act as both infrastructure as well as configuration.

Have a way to gather logs and monitor health

Things go wrong all the time and all service owners must be able to recognize precisely when things went wrong and when things righted themselves. They must also be able to gather logs to understand the root cause for the problem. It should also be possible to mine the logs and health results to identify trends. Also have a status page that enumerates the current status of your service.